Joey Hess [Fri, 21 Mar 2025 16:29:44 +0000 (12:29 -0400)]
deal with NoUUID in checkCanProxy
updatecluster, updateproxy: When a remote that has no annex-uuid is
configured as annex-cluster-node, warn and avoid writing bad data to the
git-annex branch.
The proxy.log and cluster.log end up unparseable when a NoUUID gets written
to them.
Joey Hess [Tue, 18 Mar 2025 17:55:33 +0000 (13:55 -0400)]
checkPresent of compute remote checks inputs are available
If an input file has been lost from all repositories, it is no longer
possible to compute the output. This will avoid dropping content that
was computed in such a situation, as well as making git-annex fsck --from
the compute remote do its usual thing when content has gone missing.
This implementation avoids recursing forever if there is a cycle,
which should not be possible anyway.
Note the use of RemoteStateHandle as a constructor here suggests that
this may not handle sameas remotes right, since usually a
RemoteStateHandle is constructed using the sameas uuid for a sameas
remote. That assumes a compute remote can even have or be a sameas remote.
Which doesn't seem to make sense, so I have not thought through what might
happen here in detail.
Joey Hess [Mon, 17 Mar 2025 18:27:56 +0000 (14:27 -0400)]
decided addcomputed will not support annex.smallfiles
If it did, recompute would need to somehow support recomputing
non-annexed files.
And, annex.smallfiles is typically used for configuration files or
source code kind of things, where the user doesn't want it to be an
annexed file. Computed artifacts are not likely that kind of thing.
Also, git-annex importfeed is an example of something that does support
annex.addunlocked, but does not support annex.smallfiles.
Joey Hess [Wed, 12 Mar 2025 17:36:16 +0000 (13:36 -0400)]
recompute: stage new version of file in git
When writing doc/tips/computing_annexed_files.mdwn, I noticed
that a recompute --reproducible followed by a drop and a re-get did not
actually test if the file could be reproducible computed again.
Turns out that get and drop both operate on staged files. If there is an
unstaged modification in the work tree, that's ignored. Somewhat
surprisingly, other commands like info do operate on staged files. So
behavior is inconsistent, and fairly surprising really, when there are
unstaged modifications to files.
Probably this is rarely noticed because `git-annex add` is used to add a
new version of a file, and then it's staged. Or `git mv` is used to move
a file, rather than `mv` of a file over top of an existing file. So it's
uncommon to have an unstaged annexed file in a worktree.
It might be worth making things more consistent, but that's out of scope
for what I'm working on currently.
Also, I anticipate that supporting unlocked files with recompute will
require it to stage changes anyway.
So, make recompute stage the new version of the file.
I considered having recompute refuse to overwrite an existing staged
file. After all, whatever version was staged before will get lost when
the new version is staged over top of it. But, that's no different than
`git-annex addcomputed` being run with the name of an existing staged
file. Or `git-annex add` being run with a new file content when there is
an existing staged file. Or, for that matter, `git add` being ran with a
new content when there is an existing staged file.
Joey Hess [Tue, 11 Mar 2025 16:40:21 +0000 (12:40 -0400)]
buffer responses to compute programs in a TQueue
This avoids a potential problem where the program sends several INPUT
before reading responses, so flushing the respose to the pipe could
block. It's unlikely, but seemed worth making sure it can't happen.
Joey Hess [Tue, 11 Mar 2025 15:49:47 +0000 (11:49 -0400)]
avoid error on missing compute state in checkKey
This improves eg `git-annex move --to` a compute remote that does not
contain the key. Rather than erroring with "Missing compute state" when
it checks if the key is in the remote, it proceeds to trying to store to
it, which has a nice error message.
Joey Hess [Tue, 11 Mar 2025 15:46:31 +0000 (11:46 -0400)]
add INPUT-REQUIRED
Used by git-annex-compute-singularity to make addcomputed --fast work.
Also, simplified git-annex-compute-singularity; there is no need to hard
link the container into place. singularity does not care about the
extension of the container, so can just pass it the annex object file.
Joey Hess [Mon, 10 Mar 2025 16:52:10 +0000 (12:52 -0400)]
compute: disallow output files that are not regular files
Use case where this came up is a compute program using singularity,
where the process inside the container will be allowed to write to the temp
directory, so could make eg a /etc/shadow symlink, which could then be
used to exfiltrate that from the system to wherever the annex object
might be pushed to.
It seemed better to fix this once in git-annex rather than in any such
compute program.
Joey Hess [Fri, 7 Mar 2025 18:47:34 +0000 (14:47 -0400)]
compute: add response to OUTPUT
This allows rejecting output filenames that are outside the repository,
and also handles converting eg "-foo" to "./-foo" to prevent a command
that it's passed to interpreting the output filename as a dashed option.
Joey Hess [Tue, 4 Mar 2025 18:54:13 +0000 (14:54 -0400)]
safer git sha object filename
Rather than use the filename provided by INPUT, which could come from user
input, and so could be something that looks like a dashed parameter,
use a .git/object/<sha> filename.
This avoids user input passing through INPUT and back out, with the file
path then passed to a command, which could do something unexpected with
a dashed parameter, or other special parameter.
Added a note in the design about being careful of passing user input to
commands. They still have to be careful of that in general, just not in
this case.
projects / git-annex.git / log